A brief introduction to sparklyr

What to do when code is slow?

mtcars %>% lm(mpg ~ wt + cyl, .)

# Sample
mtcars %>% dplyr::sample_n(10) %>% lm(mpg ~ wt + cyl, .)

# Profile
profvis::profvis(mtcars %>% lm(mpg ~ wt + cyl, .))

# Scale Up
cloudml::cloudml_train("train.R")

# Scale Out
mtcars_tbl %>% sparklyr::ml_linear_regression(mpg ~ wt + cyl)

Attribution: Javier Luraschi’s talk slides from SDSS 2019

Scaling Out with R and Spark

# Scale Out
mtcars_tbl %>% sparklyr::ml_linear_regression(mpg ~ wt + cyl)

Attribution: Javier Luraschi’s talk slides from SDSS 2019

How/where can you use Spark

source: Zaharia et al. (2016). Apache Spark: A Unified Engine For Big Data Processing

Example cloud platform

• Amazon EMR

Simple platform for getting started with Spark

(and sparklyr)